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NOTICE  ft 

This  document  is  disseminated  under  the  sponsorship 
of  the  U.S.  Department  of  Transportation  in  the  interest  of 
information  exchange.  The  United  States  Government 
assumes  no  liability  for  the  contents  or  use  thereof. 
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products  or  manufacturers.  Trade  or  manufacturers’ 
names  appear  herein  solely  because  they  are  considered 
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INTRODUCTION 


I 


PURPOSE- 

The  purpose  of  this  plan  is  to  describe  the  procedural  approach,  methods,  and 
responsibilities  to  be  employed  in  conducting  Operational  Test  and  Evaluation 
(OT&E)  Operational  Testing  on  the  Next  Generation  Weather  Radar  (NEXRAD)  Principal 
User  Processor  (PUP) .  This  OT&E  Operational  Testing  will  be  performed  by  the 
Federal  Aviation  Administration  (FAA)  Technical  Center,  ACW-200D,  at  the 
Washington,  DC,  and  Houston,  TX,  Air  Route  Traffic  Control  Centers  (ARTCCs)  during 
the  month  of  March  1993.  The  plan  outlines  the  method  for  evaluating  the 
operational  suitability  and  effectiveness  of  the  NEXRAD  PUP  and  aviation  weather 
products  provided  to  the  Center  Weather  Service  Unit  (CWSU)  meteorologists. 

The  OT&E  Operational  Testing  of  the  NEXRAD  PUP  will  determine  the  operational 
suitability  and  effectiveness  of  the  NEXRAD  PUP  aviation  weather  products  used  by 
CWSU  meteorologists  for  air  traffic  control  (ATC) .  The  primary  focus  will  be  to 
assess  the  usefulness  of  the  PUP's  products  as  aids  to  detecting  and  forecasting 
aviation  hazardous  weather. 

The  NEXRAD  system  is  an  "S"-band  doppler  weather  radar  system  developed  by  the 
Paramax  Corporation.  This  system  is  designed  to  detect  wind  shear  and  other 
hazardous  weather  phenomena  for  the  Department  of  Transportation  (DOT) /FAA,  and 
also  to  perform  other  functions  for  the  Department  of  Defense  (DOD)  and  the 
Department  of  Commerce  (DOC) . 


BACKGROUND 


» 


In  October  of  1983,  the  NEXRAD  Program  Council  (NPC)  requested  that  the 
Air  Force  Operational  Test  and  Evaluation  Center  (AFOTEC)  conduct  the  NEXRAD 
Initial  Operational  Test  and  Evaluation  (IOT&E).  In  order  to  accomplish  this, 
an  integrated  tridepartmental  test  team  under  the  management  of  AFOTEC  signed  a 
Memorandum  of  Agreement  (MOA)  in  April  of  1984.  The  test  team  consisted  of  the 
DOC,  the  DOD,  and  the  DOT.  The  NEXRAD  Test  and  Evaluation  Master  Plan  (TEMP) 
was  coordinated  and  approved  by  all  participating  agencies  in  March  of  1985. 

The  TEMP  detailed  the  responsibilities  of  the  participants  and  the  general  IOT&E 
scenario.  A  second  MOA  was  signed  by  the  NPC  members  and  the  AFOTEC  commander  on 
November  1988,  and  focused  on  the  I0T&E(2). 

Two  independent  test  teams  under  the  overall  management  of  the  AFOTEC  conducted 
IOT&E  of  the  competing  contractors'  (Raytheon  and  Unisys,  formerly  Sperry)  NEXRAD 
units.  Unisys  was  selected  as  the  production  contractor  and  preparations  for 
I0T&E(2)  began.  A  160-member  integrated  tridepartmental  test  team  under  the 
management  of  AFOTEC  conducted  an  I0T&E(2)  on  the  Unisys  (Paramax)  preproduction 
NEXRAD  system.  The  I0T&E(2)  was  divided  into  two  parts  (A  and  B) .  Part  A  combined 
Development  Test  and  Evaluation  (DT&E)  with  OT&E  and  contractor  activities  from 
March  1989,  to  May  1989.  Part  B  was  a  dedicated  OT&E  from  May  1989,  to  August 
1989.  The  DOC  conducted  their  part  of  the  test  at  the  Weather  Service  Forecast 
Office  in  Norman,  Oklahoma.  The  DOD  conducted  their  testing  at  Tinker  Air  Force 
Base,  Oklahoma,  at  the  Base  Weather  Station.  The  DOT  (FAA)  did  their  OT&E 
Operational  Testing  at  the  FAA  Academy  in  Oklahoma  City,  Oklahoma. 


» 
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REFERENCE  DOCUMENTS 


This  section  lists  the  applicable  documents  and  reference  materials  which  relate  to 
the  contents  of  this  plan. 

R400-SP401A  NOV  1.  1991  NEXRAD  Technical  Requirements  (NTR) 

FAA-STD-024a  Preparation  of  Test  and  Evaluation 

Documentation . 


NAS -SS- 1000  VOL  III 
NAS  ORDER  1810. 4B 
AFOTEC  PROJECT  86-0167 


NAS  System  Specification 

FAA  NAS  Test  and  Evaluation  Policy 

NEXRAD  I0T&E  PHASE  II[IOT&E(2) 
FINAL  REPORT  DECEMBER,  1989 


EVALUATION  PHILOSOPHY 


The  OT&E  Operational  Testing  of  the  NEXRAD  by  the  CWSU  meteorologists  is  designed 
to  give  them  an  opportunity  to  have  input  to  the  operational  suitability  and 
effectiveness  of  the  NEXRAD  PUP  in  an  aviation  environment.  This  input  may  be  used 
for  future  system  modifications. 

Some  of  the  features  that  the  OTfiiE  Operational  Testing  will  pay  particular 
attention  to  are  in  the  following  areas: 

*  timeliness  of  requested  products , 

*  effectiveness  of  providing  products  at  maximum  load, 

*  effectiveness  of  NEXRAD  as  an  aid  in  briefing,  consulting, 
and  coordinating  aviation  weather  problems , 

*  effectiveness  of  NEXRAD 's  automatic  alert  features  as  an  aide, 

*  effectiveness  of  NEXRAD  as  an  aide  for  the  meteorologist  in 
preparing  and  presenting  briefings  when  using  Overlays, 

Automated  Alerts ,  Time-lapse,  Dial-up,  Multiple  Screens  and 
Background  Maps, 

*  effectiveness  of  NEXRAD 's  automatic  mode  deselection  feature. 

The  results  from  this  OT&E  Operational  Testing  will  be  utilized  by  the  FAA  NEXRAD 
Program  Office  to  recommend  changes  to  the  NEXRAD,  if  necessary,  or  as  input  for 
future  systems . 


EVALUATION  APPROACH 


The  OT&E  Operational  Testing  approach  is  to  utilize  questionnaires  to  be  given  to 
the  CWSU  meteorologist.  The  operational  evaluation  will  be  conducted  at  two  sites: 
Washington,  DC,  ARTCC  and  Houston,  TX,  ARTCC. 

The  meteorologists  will  be  trained  for  the  PUP  operations  by  the  National  Weather 
Service  (NWS)  at  Norman,  OK.  CWSU  operators  are  adapting  NWS  training  of  PUP 
functionality  to  an  ARTCC  application  without  benefit  of  an  FAA  course.  They  will 
receive  4  weeks  of  training  on  the  PUP  operational  activities  and  the  interpreta¬ 
tion  and  dissemination  of  NEXRAD's  hazardous  weather  information.  Additional 
on-the-job  training  will  be  conducted  at  the  existing  PUP  site.  The  FAA  Technical 
Center,  ACW-200D  representative  will  brief  the  meteorologists  on  the  purpose, 
scope,  and  procedures  of  the  evaluation  and  distribute  the  questionnaires. 

The  meteorologist  will  complete  the  questionnaire.  The  questionnaire  will  be 
organized  to  address  the  effectiveness  of  NEXRAD  as  an  aid  in  performing  specific 
aspects  of  the  meteorologist's  job  in  the  CWSU.  Following  the  evaluation,  the 
completed  questionnaires  will  be  analyzed  by  ACW-200D  personnel.  The  data 
collected  will  form  the  basis  for  a  draft  report,  which  will  be  forwarded  to  the 
principal  organizations  supporting  the  NEXRAD  PUP  OT&E  Operational  Testing.  A 
final  report  will  follow  the  draft  report. 


EQUIPMENT 


The  PUP  receives  NEXRAD  products  from  the  Radar  Product  Generator  (RPG)  upon 
request  from  the  operational  personnel.  The  PUP  contains  the  display  unit  for 
product  control,  status  monitoring,  local  annotation,  and  product  archiving.  The 
PUP  has  three  major  components:  the  data  processor,  the  system  console,  and  the 
workstation. 

The  data  processor  is  a  two-bay  cabinet  containing  the  necessary  hardware/software 
for  all  data  processing  activities.  The  cabinet  also  contains  the  narrow-band 
links  used  between  the  PUP  and  its  associated  Radar  Product  Generator  (RPG)  for 
internal  product  request  and  distribution,  and  for  external  requests  and 
distribution  from  nonassociated  RPGs .  The  NEXRAD  PUP  has  one  dedicated  port  to  its 
associated  RPG  and  four  dial-up  ports  for  access  to  four  additional  RPGs  at  any 
location  in  the  radar  network.  Each  RPG  has  two  dedicated  ports  and  two  dial-up 
ports  assigned  for  an  FAA  product  request. 

The  system  console  holds  an  alphanumeric  terminal  consisting  of  a  keyboard  and  a 
14-inch  cathode  ray  tube  (CRT)  screen.  The  console  is  an  interactive  peripheral 
device  connected  to  the  display  processor  port  on  the  data  processor  cabinet.  The 
functions  provided  by  the  console  include  system  initialization  and  startup, 
control  of  the  disk/tape  operations,  loading  and  execution  of  processor  diagnostic 
programs,  and  start/stop  of  the  systems  application  software. 
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The  workstation  is  the  primary  operational  position  for  the  PUP.  It  provides  the 
operator  with  the  capability  to  request,  display,  and  manipulate  weather  products 
generated  by  the  RPG.  The  workstation  consists  of  a  table  structure  with  operator 
work  space  and  support  provisions  for  an  application  terminal,  two  color  monitors, 
a  graphic  tablet,  and  an  audio  alarm.  A  hard  copy  device  is  also  provided  on  a 
separate  table  that  may  be  positioned  up  to  50  feet  from  the  workstation. 

The  PUP  is  connected  to  the  NEXRAD's  RPG  unit  by  narrow-band  links  within  the 
communication  group  of  the  NEXRAD  Communication  Group.  This  connection  is  used  for 
internal  product  request  and  distribution,  for  interunit  product  request  and 
distribution,  and  for  distribution  of  products  to  other  users.  The  links  are 
interconnected  for  transmission  over  one  direct  line  and  other  dial-up  commercial 
telephone  lines  for  external  data  requests. 

ORGANIZATION  ROLES  AND  RESPONSIBILITIES 


The  principal  organizations  participating  in  the  OT&E  of  the  PUP  include  FAA  NEXRAD 
Program  Office  (ANR-500),  FAA  Technical  Center  (ACW-200D),  and  Washington  and 
Houston  ARTCCs. 

RESPONSIBILITIES. 


ANR-500.  ANR-500  has  the  overall  responsibility  for  the  NEXRAD  PUP  Acquisition  and 

Deployment . 

FAA  Technical  Center/ACW-200D.  The  FAA  Technical  Center,  ACW-200D,  has  been 
designated  to  provide  the  Associate  Program  Manager  for  Test  (APMT) .  The  APMT  is 
responsible  for  the  overall  management  and  direction  for  the  NEXRAD  PUP  OT&E 
Operational  Testing.  ACW-200D  will  complete  and  publish  an  operational  evaluation 
Test  Plan  and  associated  questionnaires,  make  initial  site  visits,  and  coordinate 
with  the  CWSU  to  ensure  that  the  PUP  site  will  have  CWSU  meteorologists  available 
to  successfully  complete  the  OT&E  Operational  Testing.  ACW-200D  will  also 
administer  the  questionnaires  to  the  CWSU  meteorologist.  An  analysis  will  be  based 
on  data  collected  from  the  responses  to  the  questionnaire.  This  analysis  will  be 
used  in  the  preparation  of  a  draft  report  and  a  final  report  from  ACW-200D. 

Washington  and  Houston  ARTCC  CWSUs.  The  CWSUs  will  provide  the  ACW-200D  Evaluator 
with  access  to  the  CWSU  meteorologists  that  will  participate  in  the  OT&E 
Operational  Testing. 

E8LES • 

The  roles  for  conducting  the  NEXRAD  PUP  OT&E  Operational  Testing  are  as  follows: 

AMPT.  The  AMPT  is  responsible  for  the  overall  management  and  direction  for  the 
NEXRAD  PUP  OT&E  Operational  Testing  as  detailed  in  FAA  Order  1810. 4B. 
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EVALUATOR .  The  Evaluator  Is  the  ACW-200D  representative  who  will  administer  and 
conduct  the  on-site  evaluation  of  the  NEXRAD  PUP  at  the  Washington  and  Houston 
ARTCCs.  The  Evaluator  will  participate  in  all  site  visits,  reviews,  and  briefings. 
The  Evaluator  will  perform  data  analysis  and  prepare  the  reports. 

EVALUATION  PARTICIPANTS .  Three  Houston  and  four  Washington  CWSU  meteorologists 
will  evaluate  the  NEXRAD  PUP  by  completing  the  associated  questionnaire. 


DOCUMENT  REQUIREMENTS  AND  CONTROL 


The  documents  required  to  conduct,  describe,  and  report  the  results  of  the  OT&E 
Operational  Testing  of  the  NEXRAD  PUP  are  listed  below. 

EVALUATION  PLAN. 

This  plan  describes  the  requirements,  methods,  and  responsibilities  for  conducting 
the  OT&E  Operational  Testing  of  the  NEXRAD  PUPs  located  at  the  Washington  and 
Houston  ARTCCs.  It  provides  a  plan  that  will  allow  for  an  evaluation  by  the  CWSU 
meteorologist  in  a  live  operational  environment. 

EVALUATION  QUESTIONNAIRE . 

A  questionnaire  will  be  developed  by  ACW-200D  in  order  to  obtain  feedback  from  the 
PUP  CWSU  meteorologists.  The  questionnaire  will  focus  on  the  overall  suitability 
and  effectiveness  of  the  PUP  as  an  aid  for  the  CWSU  meteorologists  in  preparing 
accurate  and  timely  aviation  weather  warnings.  The  questionnaire  will  be 
structured  to  obtain  the  evaluation  by  rating  a  statement  about  each 
feature/function  on  a  five-point  scale  ranging  from  Very  Good  to  Very  Poor. 
Comments  will  also  be  encouraged. 

TEST  REPORT  SUMMARY. 

A  Test  Report  Summary  will  be  submitted  to  ANR-500  within  15  days  from  completion 
of  the  test. 

FINAL  REPORT- 

ACW  200D  will  prepare  the  final  report  and  will  include  conclusions,  problems/ 
issues,  and  recommendations  resulting  from  the  evaluation.  This  report  will  be 
sent  to  ANR-500,  and  the  participants,  by  May  15,  1993. 
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TRAINING 


Detailed  training  of  the  ARTCC  CWSU's  meteorologists  participants  will  be  conducted 
by  the  NWS  through  the  Operational  Support  Facility  (OSF)  at  Norman,  OK,  in 
coordination  with  the  Houston  and  Washington  ARTCC  CWSUs . 
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EVALUATION  SUPPORT  REQUIREMENTS 


This  section  describes  the  instrumentation  required  for  the  evaluation,  the  types 
of  data  analysis  to  be  performed,  and  the  evaluation  configuration. 

INSTRUMENTATION. 

No  special  instrumentation  is  required  for  the  evaluation.  The  PUP  is  presently 
installed  at  the  Washington  ARTCC  CWSU  with  all  of  the  necessary  equipment.  The 
PUP  will  be  installed  at  the  Houston  ARTCC  starting  the  week  of  January  11,  1993. 
The  two  PUPs  will  be  similar  in  their  configuration. 

DATA  ANALYSIS. 

Numerical  values  will  be  assigned  to  the  questionnaire  response,  and  then 
statistically  analyzed.  The  intent  of  the  analysis  Is  to  find  the  degree  of 
satisfaction  or  dissatisfaction  the  CWSU  meteorologists  have  with  the  NEXRAD  PUP. 
The  suggestions  and  open  comments  will  be  analyzed  using  content  analysis.  These 
comments  will  have  higher  weight  in  the  analysis  due  to  the  small  amount  of 
respondents.  The  responses  will  be  grouped  so  as  to  provide  maximum  meteorologist 
feedback.  The  evaluation  of  the  responses  to  the  questionnaires  will  help  to 
answer  questions  regarding  CWSU  operational  usage  of  the  NEXRAD  PUP. 

EVALUATION  CONFIGURATION. 

The  evaluation  is  taking  place  at  the  Washington  and  Houston  ARTCC  CWSUs.  Each  of 
the  NEXRAD  PUP  installations  is  expected  to  include  one  dedicated  port  to  the 
associated  RPG,  one  dial -out  port  to  the  RPG,  one  dedicated  port  to  the  PUP 
external  system,  and  one  dial-in  port  from  other  users.  The  installations  will 
also  contain  the  hardware/software  required  for  graphics  processing,  local  control, 
status  monitoring,  local  annotation,  product  archiving,  and  communication  narrow- 
band  links.  A  system  console  supports  an  alphanumeric  terminal  consisting  of  a 
keyboard  and  a  14 -inch  CRT  screen.  The  system  console  provides  for  the 
initialization  and  startup,  control  of  the  disk/tape  operations,  loading  and 
execution  of  the  processor  diagnostic  programs,  and  start/stop  of  the  system 
application  software.  The  workstation  is  the  primary  operational  position  for  the 
NEXRAD  PUP  Unit.  It  contains  two  graphic  displays,  an  alphanumeric  display,  a  hard 
copy  device,  and  video  interface  connections. 

The  Washington  ARTCC  PUP  has  been  installed  with  the  above  configuration.  The 
associated  NEXRAD  site  for  the  Washington  PUP  is  located  in  Sterling,  VA.  The 
Houston  ARTCC  PUP  is  to  begin  installation  the  week  of  January  11,  1993.  The 
associated  NEXRAD  site  for  the  Houston  ARTCC  PUP  is  located  in  League  City,  TX. 


REVIEWS  AND  MEETINGS 


The  visits,  briefings,  and  reviews  described  below  are  necessary  to  successfully 
conduct  the  OT&E  Operations  evaluation  of  the  NEXRAD  PUP. 


> 


INITIAL  SITE  VISIT. 

An  Initial  visit  to  the  Washington  ARTCC  CWSU  was  needed  to  obtain  specific 
information  necessary  to  conduct  the  evaluation.  Information  that  was  obtained 
included  the  number  of  meteorologists  that  will  be  participants,  layout  of  the  PUP 
area,  display  equipment  being  used,  duty  schedule  of  participants,  etc.  The 
information  from  this  initial  visit  to  the  Washington  ARTCC  CWSU  will  suffice  for 
both  the  Washington  and  the  Houston  ARTCC  CWSUs  sinc°  both  sites  are  similar. 

EVALUATION  BRIEFING. 

Briefings  will  be  held  at  the  CWSU  to  insure  that  the  intent  of  the  evaluation  is 
conveyed,  the  logistics  are  understood,  and  the  questionnaires  are  thoroughly 
explained.  The  FAA  Technical  Center  ACW-200D,  who  will  be  conducting  the  briefing, 
will  insure  tlat  each  participant  has  been  briefed  and  that  all  questions  and 
concerns  are  addressed.  The  briefing  will  be  held  on  an  as  needed  basis  so  that 
all  duty  shifts  are  covered. 


SCHEDULE 


Initial  visit  Washington  ARTCC/CWSU 
Develop  Evaluation  Plan 
Prepare  Evaluation  Questionnaires 
Operational  Evaluation  Complete 
Test  Summary 
Final  Report 


November  19 ,  1992 

December  1992,  through  January  1993 
January  1993,  through  February  1993 
March  1993 

15  days  after  the  evaluation 
May  15,  1993 


> 


NEXPAP  PVP  EVALUATION  REQUIREMENTS  MATRIX- 

THE  NEXRAD  PUP  requirements  are  contained  in  the  NEXRAD  Technical  Requirements 
(NTR)  and  in  the  National  Kir  Space  (NAS)  SS-1000.  These  requirements  will  be 
utilized  to  insure  traceability  from  the  NEXRAD  OT&E  Operational  Test  question¬ 
naires  to  the  NAS -SS- 1000.  A  NEXRAD  PUP  OT&E  Operational  Test  matrix  has  been 
developed  to  contain  the  specification  references,  requirements,  and  the  evaluation 
questionnaire  category  under  which  it  will  be  evaluated  (appendix  A) . 


» 


» 


» 
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AFOTEC 

ACRONYMS  AND  ABBREVIATIONS 

Air  Force  Operational  Test  and  Evaluation  Center 

1 

APMT 

Associate  Program  Manager  for  Testing 

ARTCC 

Air  Route  Traffic  Control  Center 

ATC 

Air  Traffic  Control 

1 

CRT 

Cathode  Ray  Tube 

CWSU 

Center  Weather  Service  Unit 

DOC 

Department  of  Commerce 

» 

DOD 

Department  of  Defense 

DOT 

Department  of  Transportation 

DT&E 

Development  Test  and  Evaluation 

» 

FAA 

Federal  Aviation  Administration 

IOT&E 

Initial  Operational  Test  and  Evaluation 

MOA 

Memorandum  of  Agreement 

1 

NAS 

National  Airspace  System 

NEXRAD 

Next  Generation  Weather  Radar 

NPC 

NEXRAD  Program  Council 

1 

NTR 

NEXRAD  Technical  Requirements 

NWS 

National  Weather  Service 

OSF 

Operational  Support  Facility 

> 

OT&E 

Operational  Test  and  Evaluation 

PUP 

Principal  Users  Processor 

RPG 

Radar  Product  Generator 

» 

TEMP 

Test  and  Evaluation  Master  Plan 

► 
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APPENDIX  A 


N EXRAD  ARTCC  CWSU  METEOROLOGIST 
OT&E  OPERATIONAL  MATRIX 


N EXRAD  ARTCC  CWSU  METEOROLOGIST 
OT&E  OPERATIONAL  MATRIX 


NAS -SS- 1000 
VOLUME  III 

REQUIREMENT  DESCRIPTION 

QUESTIONNAIRE 

SECTION 

REMARKS 

1.2. 1.2. 4. 1.5 

Distribution  of  WX 
products . 

section  I 

1.2. 1.2. 4. 1.7 

Accept  and  process 
operational  control. 

section  I 

!  2. 1.2. 4. 1.8 

Accept  and  process  remote 
maintenance  control. 

section  I 

N EXRAD  TECHNICAL 
REQUIREMENTS  (NTR) 


3.6 

Training 

section  III 

3.7.3 

PUP  functional  area 
requirements . 

section  II 

3. 7. 3.1 

PUP  display  and  data 
entry. 

section  II 

3. 7. 3. 2 

Product  request. 

section  I 

3. 7. 3. 3 

Product  display. 

section  I 

3. 7. 3. 4 

Local  product  storage. 

section  I 

3. 7. 3. 5 

FAA  Product  annotation 
and  distribution. 

section  I 

3. 7. 3. 6 

PUP  control. 

section  II 

3. 7. 3. 7 

Status  monitoring 

section  III 

3. 7. 3. 8 

Archiving 

section  III 

